While malicious software certainly targets traditional computers running a single operating system, it can also target a computer running multiple operating systems. In a virtualization environment, different operating systems may run on a single computer and these may be subject to unique types of malicious software.
FIG. 1 illustrates a prior art virtualization environment 10 that is subject to attacks by malicious software. Any suitable computer hardware 20 executes a virtualization platform 30 which is a layer of software running directly on the computer hardware and which replaces the traditional operating system. The platform 30 allows the computer hardware to execute multiple operating systems concurrently such as a Microsoft operating system 50, a Linux operating system 60, Solaris, NetBSD, FreeBSD, and others. The privileged domain 40 may execute under any of a variety of operating systems as well. Each operating system 40-60 then may execute independently of the others and therefore each is considered a virtual machine (VM).
Unfortunately, the nature of a virtualization environment allows for a new threat called an inter-VM attack, in which malicious software under one operating system attacks programs and data under another operating system executing on the same host computer. An attack can also take place between virtual machines—from one host computer to another host computer.
Inter-VM attacks can be especially problematic in a public virtual cloud environment. Traditional network security software not accustomed to a virtualization environment has difficulty detecting or containing malicious inter-VM traffic between the virtual machines. More and more, a great deal of data center network traffic occurs between virtual machines on a host computer server, but, administrators find it more and more difficult to monitor such virtual machine traffic or to implement inspection or filtering policies. Such traffic between virtual machines might be invisible to traditional network monitoring tools such as packet inspection or filtering because such traffic does not use the physical network.
While one approach might be to scale back any virtualization efforts, this defeats the promise that virtualization offers terms of economic benefits. Physical security products cannot detect attacks that go from one virtual machine to another on the same host computer. And, attempting to send all inter-VM traffic out to the network in order to detect these inter-VM attacks is undesirable because it increases network latency. Accordingly, new techniques are desired to address inter-VM attacks and other malicious software within a virtualization environment.